1. Field of the Invention
The present invention relates to a frame interpolation technique for generating a nonexistent video frame between two video frames by using interpolation.
2. Description of the Related Art
For a liquid crystal display (LCD), which is one of displays used for flat-screen televisions, a hold display for continuously drawing an image until the next video frame reaches is used in contrast to an impulse display for drawing a momentary image on a display such as a CRT (Cathode Ray Tube) or a plasma display.
With the hold display, the phenomenon of motion judder is caused, in a moving object being displayed, by a mismatch between motion interpolation according to pursuit eye movements and the hold display for remaining the position of the moving object unchanged. Additionally, with ISDB-T one-segment broadcasting, which has started in recent Japan, a motion is noticeably unnatural in some cases because a frame rate is on the order of 15 fps, which is lower than the current analog broadcasting.
As a solution to this problem, it is effective to generate an intermediate frame between video frames, and to present a video image the motion of which is interpolated. How to make the motion interpolation is depicted in FIG. 1A. An interpolation frame 102, for example, exactly at a midpoint time t−½ is generated by obtaining a motion vector within a screen from a frame 101 at a time t−1, and a frame 103 at a time t, and by compensating for a motion after a time of ½. This can improve the quality of the video motion.
However, a motion vector search requires an enormous amount of calculation. Especially, if a large motion, namely, a video image the frame interpolation effect of which becomes remarkable is interpolated, a search cannot be terminated in realistic time. Accordingly, frame interpolation of a scene screening type, with which frame interpolation is implemented by interpolating only a scene the motion of which has a particular feature, has been proposed as a practically effective method (for example, see the following Patent Documents 1 and 2).
Patent Document 1: Japan Patent Application Publication No. 11-112940
Patent Document 2: Japan Patent Application Publication No. 2000-134585
FIG. 1B shows a configuration example of such a frame interpolating apparatus of a scene screening type. A scene screening unit 201 determines whether or not a scene is one having a feature, according to a motion vector obtained by a motion vector searching unit 202, and outputs an interpolation control signal to an interpolation image generating unit 203. The interpolation control signal becomes ON (valid) if a scene having a feature is detected, or becomes OFF (invalid) unless otherwise detected. The interpolation image generating unit 203 generates an interpolation frame from a motion vector only if the interpolation control signal is ON.
Examples of a scene the motion of which has a particular feature include a constant-speed scrolling scene. In this case, it is necessary to satisfy a condition that the directions of vectors within a screen are identical. However, if a portion not in motion, such as a superimposed telop, exists within a screen, the portion is not included by scene screening due to a 0 component of the corresponding vector in some cases. Additionally, motion vectors vary in a portion such as the sky or a water surface where the image complexity is low, leading to difficulty in scene screening. If the scene screening cannot be made, unnaturalness as a moving image becomes noticeable on the contrary due to a switching between the presence and the absence of an interpolation frame.
Furthermore, in a portion, such as a window of a building, where a artificial pattern is repeated, a motion vector tends to be erroneously searched, and such a portion sometimes becomes noticeable as a portion degraded by erroneous interpolation when an interpolation frame is generated.
It is conventionally pointed out that the accurate obtainment of a motion vector is important in frame interpolation, and a technique using a prediction vector is known as one of methods for obtaining a motion vector. The above cited Patent Documents 1 and 2 disclose a method for predicting and searching a motion vector from within a space.
With this prediction technique, a motion vector in a medium region 302 is searched based on a result of a search made in a large region 301, and a motion vector in a small region 303 is searched based on a result of the search made in the medium region 302 as shown in FIG. 1C. With this method, motion vectors are allowed to vary while a search is again made in a smaller region to satisfy a condition of the minimum prediction residual if a prediction error is large.
In contrast, also a technique for predicting a motion vector not in a spatial direction but in a temporal direction has been proposed (for example, see the following Patent Document 3).
Patent Document 3. Japan Patent Application Publication No. 2006-331136
FIG. 1D shows a configuration example of a frame interpolating apparatus using a prediction vector in a temporal direction. A motion vector obtained by a motion vector searching unit 401 is held in a motion vector memory 402 as a reference motion vector, and used to search a motion vector of the next frame. An interpolation image generating unit 403 generates an interpolation frame from the motion vector.
With this method, there are no problems if a prediction vector in a temporal direction is suitable. However, an undesirable motion is sometimes generated especially in a portion, such as the sky or a water surface, where an activity is low. In such a portion, it is difficult to determine whether or not a motion vector is proper regardless of the direction of the motion vector. Therefore, a search result is confined to a prediction vector given in a temporal direction even if the prediction vector differs from a proper motion vector. As a result, an erroneous motion vector continues to be selected, leading to degradation in the video image of an interpolation frame.
Furthermore, since prediction in a spatial direction is not made, this method is totally ineffective for the above described variance of motion vectors.
The following Patent Document 4 relates to an interpolation frame generating apparatus that can display a background image more sharply by assigning an accurate motion vector to an occlusion region.
Patent Document 4: Japan Patent Application Publication No. 2007-060192